DataQuest Capstone: Exploratoray Data Analysis of Reader Engagement By Book Genre¶

Hypothesis and Approach¶

The goal of this project is to make a recommendation for the launch of an online book group. We want to know which types of books tend to lead to the most engaged readers. The books should both appeal to a large volume of readers and tend to drive discussion among readers, since these qualities would support the group's success.

From an informal poll of my network I've formed a hypothesis that fiction books would lend themselves better to an online book group than non-fiction. Now I want to examine a large dataset that will either lead me to support or reject that hypothesis. I also want to learn if there are more specific categories of books that would lead to more interest.

I'm using a dataset from Kaggle that compiles data for 100k books scraped from Goodreads. Goodreads is an extensive database of books that allows users to rate and review books, and to generate reading lists. As of 2022 the site reported 90 million registered members so we can assume that it reaches a broad audience of book readers.

We're making a couple of important assumptions:

• That the dataset of 100k books from Kaggle is representative of the total population of books on the site.
  
  
• That the behavior of users on Goodreads is a good proxy for readers' inclination to engage with an online book group. Specifically, we'll be looking at the number of star ratings the book received as an indication of how widely read it is, and the number of reviews users wrote, as an indication of how much discussion the book elicited.

Data Exploration¶

First we'll import the Python libraries, then load the csv file into a dataframe, view a few rows, and generate some descriptive stats about the data. Next we'll determine which columns to focus on and what steps we should take to clean the data.

These are the findings from the initial exploration of the data:

• There are 100k rows and 13 columns in the dataset.
  
  
• The genre column contains information about whether the book is fiction or non-fiction as well as other genres.
  
  
• Numeric columns include pages, rating, reviews, and totalratings. All of these may be useful in our analysis, except for pages, because we don't care how long the book is.
  
  
• Books tend to have far more ratings than reviews. It likely takes more effort to write out a review than to leave a rating. Both of these columns are important for our analysis.
  
  
• The star rating indicates how satisfied users are with the book, which could also be important. The max rating is 5, and the average for all books is 3.8, which is fairly high. It's possible users are more inclined to rate books they liked, or our dataset is skewed. This may be less important to us than the number of reviews and ratings, but we will consider it in the analysis.
  
  
• 25% of the dataset had fewer than 31 ratings. This is important because if a book had very few ratings the average rating might be easily skewed.

Let's look at the total and unique values in the non-numeric columns and determine how we should clean the data.

These are the findings around data cleaning:

• While the numeric columns don't have missing values, most of the non-numeric columns have some missing values.
  
  
• There are 72k unique values in the genre column because each book has multiple genres contained in a string, so these will need to be broken apart for them to be useful. The genres are generated from the shelf names of users who add the the book to their shelves. We can expect there to be overlap in the shelf names, and one book may be categorized under many different genres. A fair amount of cleanup and filtering will be needed here.
  
  
• There are repeated values for author, but this is to be expected since authors can publish multiple books.
  
  
• There are only 725 unique values for ISBN13. ISBN13 should be unique to the title and format which is important for determining whether there are duplicates in this dataset. Therefore, ISBN13 probably won't be very useful, but the ISBN column (which is the 10-digit version of the ISBN) had far more unique values (86k) and no repeat values although there are some missing values.
  
  
• There are a lot of book formats (202). While this isn't a primary focus of our analysis it could prove useful, so we'll try to clean these up.
  
  
• There are 2,411 duplicate titles and one missing title. We should see if these are truly duplicates or could they be multiple formats of a single book.

Data Cleaning¶

Now that we know how to clean the data, we'll drop the columns we don't need, reorder the remaining columns, and then see if there are any duplicate titles. Because we have some missing values in the ISBN column, we'll consider an entry to be potentially duplicated if the title, author, ISBN and format all match.

These look like duplicate titles, but they have different numbers of ratings and reviews. Perhaps they were scraped at different times. Let's remove the duplicates, and keep the entry with the most ratings.

Which book is missing a title?

Since neither the title nor the ISBN is available on Goodreads we'll remove this from the dataset.

These two steps eliminated 62 rows. Now let's take a look at what the book formats are.

We'll clean these up by changing the case to lower for consistency, grouping some of the values together (like "mass market paperback" and "trade paperback") and grouping some of the little-used formats into an "other" category.

We can see that paperbacks are by far the most common format on the platform, but hardcovers are significant. Ebooks, Audiobooks, and other formats are far less common.

Now we'll move on the cleaning the genre column. First, we have a number of null values to consider.

There are 10,467 rows that don't have the genre filled in at all, so we don't know if those books are fiction or non-fiction. We'll fill these rows with "no genre" so they will be considered as a group in the final analysis. If we see high reviews and high ratings for this group it may be worth doing further exploration to see if we can get genre information for these titles.

Next, we'll do the following steps:

• Change the genre column to all lowercase to eliminate any duplication due to inconsistent capitalization.
  
  
• Create a column called new_genre that copies the genre column.
  
  
• Clean up some instances in new_genre where "non fiction" includes a space.
  
  
• Create a new column called supergenre that specifies whether the title is fiction or non-fiction. If "nonfiction" appears in the new genre column it will be labeled "non-fiction" with a hyphen.
  
  
• Calculate how many books are non-fiction, or "no value."

Now add some logic so if "fiction" appears anywhere in new_genre and is not preceded by "non" the supergenre will be "fiction" (such as in the case of "historical fiction").

Adding the fiction categorization reduced the number of non-fiction books from 29,479 to 28,398. So it appears that there are some books that have both "fiction" and "non-fiction" in the new_genre column. Let's take a closer look at what those are.

We see that these books include biographies, military history, self-help and other topics we normally find in the non-fiction section, so we'll relabel these non-fiction.

We can see that for about a third of our books we don't know if it should be categorized as fiction or non-fiction. We'll have to see if we can categorize them based on other information in the genre column. First we need to transform the new_genre column from strings to lists. Then we'll explode the new genre column which will create one row for every genre in the list. Consequently, a single book can have multiple rows corresponding to each genre listed in its new genres column.

Let's verify that this worked by checking a single ISBN in the old and the new dataframe. In the old data we expect to see one row per ISBN.

In the new dataframe we expect to see 9 rows, one for each genre in the new_genre column, which is what we're seeing.

We can now quickly see if some books had duplicate genres and remove duplicate rows, so that each book, format, author and ISBN has only one row for each applicable genre.

This step removed 152,263 rows. Now let's see how many fiction and non-fiction books we have, and how their average ratings compare, by aggregating data in our unexploded dataset (which has one entry per book).

There are a lot of books with "no value" for the supergenre which means it didn't have a value of "fiction" or "non-fiction" in the new_genres column so we don't know how to classify it. Let's see if we can classify some of these by looking at what the most common genres are when the supergenre is "no value."

We'll remove the "no genre" books from our dataset because we don't have a way to get their genre, they only averaged about 1 review per title. We're most interested in the books that have high reviews. We'll remove all children's and picture books from the dataset since our group won't be for children. Other genres we can verify by checking them here. Anything with a genre of fantasy, sequential art, comics, manga, graphic novels, we'll categorize as fiction. And if the genre contains romance we'll also categorize these as fiction, because the romance genres appear under fiction on the Goodreads genre list.

Now that we've categorized the books by supergenre as best we can and cleaned up the genre column we can move on to the final analysis and visualizations.

Data Summary: Fiction vs Non-Fiction¶

Let's visualize our cleaned-up dataset to see whether fiction or non-fiction books tend to get more engagement, which we'll quantify by looking at the number of reviews per title for each.